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ABSTRACT 



The research methods selected for an evaluation of courses 
in vocational education and training (VET) teacher education at Curtin 
University (Western Australia) were examined. The university hired an outside 
research body from a different TAFE college to conduct the evaluation because 
Curtin University’s courses in postschool teacher education, designed for 
trainers in Technical and Further Education (TAFE) colleges, industry, and 
government, were reviewed and extensively restructured 3 years earlier, and 
the teaching staff considered themselves too close to the program. Planned 
evaluation methods will include a review of the literature, including state 
and federal government policy directives; interviews with TAFE and industry 
management; interviews with Curtin lecturers; and surveys of past and present 
students. An in-depth analysis and discussion of results should assist the 
evaluators to come up with a set of recommendations to be published in a 
detailed report. It is expected that after the evaluation is completed and 
any recommendations implemented, Curtin University will have strong evidence 
to continue to claim that its courses are relevant, up-to-date, and based on 
industry training needs. (KC) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Evaluating teacher training for industry trainers 



Clare McBeath 

Curtin University, Western Australia 

Lindsay Nicholson 

QRD consulting, West Coast College 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



£ 




1 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



This paper examines the selection of research methods for an evaluation 
of Curtin University's courses in vocational education and training (VET) 
teacher education. 




1 1 c; DFPARTMENT OF EDUCATION 

PAQfiarrh and Improveme 



The evaluation will be carried out by an outside research body. As the 
courses were newly upgraded and rewritten for distance and online 
delivery just over three years ago, the lecturing staff are still too close to 
the material to undertake a new evaluation. The selection of a research 
body from a technical and further education (TAFE) college, in this 
instance, will provide impartiality towards the material and a more valid 



□ Minor changes have been made to 
improve reproduction quality. 



originating it. 




official OERI position or policy. 



assessment of market need. 

Methods will include a review of the literature, including State and 
Federal government policy directives, interviews with TAFE and industry 
management, and with Curtin lecturers and surveys of past and present 
students. 



It is a requirement at Curtin University, as at many other educational institutions, to 
review courses every five years. The review process is an accountability procedure to 
ensure that courses are regularly updated and their relevance evaluated against the 
needs of the students and changing practices in society and industry. 

Curtin University's courses in post-school teacher education, designed for trainers in 
TAFE colleges, industry and government, were reviewed and extensively 
restructured three years ago. There were then five vocational education and training 
courses offered at undergraduate and graduate level, as follows: 

• Associate Degree (Training) 

• Bachelor of Arts (Vocational Education and Training) 

• Graduate Certificate (Tertiary Teaching) 

• Graduate Certificate (Training) 

• Graduate Diploma in Education (Higher and Further) 

The current courses had been revised using data from surveys of past and present 
students and industry employers, as well as incidental feedback from students to 
individual lecturers. The course team had also appraised prospectuses from other 
universities offering similar courses. The selection of units (or subjects) to be 
included in the courses was rigorous. Each unit had been individually justified and 
reshaped according to the survey data before it was accepted by a critical curriculum 
decision making team. - 
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BEST COPY AVAILABLE 



The restructured courses also had to be adapted to fit new university requirements 
on credit points, duration and unit size. They had to be written for distance 
education and reformatted for offering online. It was possibly the most detailed 
update since the course was introduced over 20 years ago. The upgrade was a very 
heavy undertaking for all those involved, demanding a high level of professional 
energy to work intensively on so many subjects at one time, and the development 
team members were anxious to move on with other aspects of their professional lives 
when the upgrade was finished. 

Now the courses are entering their fourth year and it is necessary to begin the review 
process again, to evaluate again in time for rewriting in 2002 and for offering the next 
upgrade in 2003. 

This announcement was greeted initially with disbelief. It seemed such a short while 
since the last upgrade was completed. The staff began to articulate their fear that 
they had very little new to offer to their courses. The recent, intensive rewrite had left 
them very close to their own units and drained of new ideas. They had put as much 
as they could into the materials and it was easy to feel that they could not be 
perfected any further. 

Meanwhile, as we know, the training scene itself has changed. There has been a 
general downgrading of desired professional qualifications in the training scene. 
University degrees are undervalued by TAFE colleges, the State Departments 
responsible for training and employment and by ANTA on the federal scene. Very 
few trainers receive either time or financial assistance to undertake university studies 
any more. The exceptions to these constraints are coming from industry, but industry 
trainers are more difficult to reach and their numbers in Curtin's courses are still 
small. 

Economic rationalism has brought about changes in the language of training. An 
overemphasis on training as little more than the assessment of observable 
competencies has encouraged a minimalisation of reflective practice and critical 
thinking. As much as we may deplore this, we still have to be aware of its existence 
and make sure our courses acknowledge the strengths and weaknesses of such 
approaches. 

Furthermore, industry reform and the devaluing of university professional 
qualifications have made it much harder to market university courses for trainers. 
Student numbers are falling across Australia. In these circumstances, we certainly 
need to review our courses and see what we can do to make them more attractive to 
the market for which they were designed. 

Why evaluate courses 

Virtually all the literature on curriculum development emphasises the importance of 
evaluation as part of the course development process. However, it is probably the 
most neglected part of development. Often this is because funded development 
tends to use up the budget on needs analysis and on the production of course 
materials, and evaluation of the completed materials is dropped for economic 
reasons. Or, evaluation may take place during the writing and trialling of the 
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materials, but rarely as a formal process once the materials have been used for a 
number of years. 

Even in the university context, where the five-year review and evaluation process is 
supported by policy and practice, more often than not, the process is not specifically 
funded and staff are rarely given release time to do it properly. Sometimes an 
outside reviewer is called in as part of the process and some of the more important 
decisions are left in his or her hands, but most of the work usually remains the 
responsibility of overworked staff. 

Yet there are many reasons why courses should be regularly reviewed and updated. 
Basically, it is to make them more saleable in a highly competitive field, but there is 
also an intrinsic satisfaction in being able to say that one's own course materials are 
current, relevant, of high quality, well structured and well written. I suppose most 
course coordinators would like to think that their own coursework might be the best 
in the country - or at least that is how we feel. 

Tyler, reviewing the evaluation field in 1991, listed six reasons for evaluation, 
namely: 

1. to monitor present programs; 

2. to select a better available program to replace one now in use that is deemed 
relatively ineffective; 

3. to assist in developing a new program; 

4. to identify the differential effects of the program with different populations of 
students or other clients; 

5. to provide estimates of effects and costs in the catalogue of programs listed in 
consumer resource centres; 

6. to test the relevance and validity of the principles upon which the program is 
based (Tyler 1991, p 4). 

Points 1, 4 and 6 probably apply best to our planned evaluation. We want to monitor 
it. We believe it is already a good program and we do not intend to develop a new 
one. We hope to tap into the client groups and their management and check whether 
we are meeting their perceived needs. We are already locked into university policies 
regarding funding and cost effectiveness, for better or for worse. Most importantly to 
us, however, is Tyler's final point on testing the relevance and validity of the current 
program and making changes if and as necessary. 

Specifically, the purpose of our evaluation is to find out the worth or value of the 
courses in a changing environment. Curriculum teams need to find out how good 
their courses and materials are, and whether they continue to work in practice, or 
whether parts have to be changed or adapted. 

We need to go far beyond verifying that the course objectives match the course 
outcomes, as some evaluators claim is what is required. Educational programs may 
have many aims, many objectives, many outcomes not expressed in the aims and, 
often, many stakeholders or audiences with different aims (Straton 1975). Straton saw 
evaluation as follows. 
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Educational evaluation is the process of delineating, obtaining, and 
providing information about an educational programme which is of use in 
describing the programme and in making judgments and decisions related 
to the programme (Straton 1975, p 4). 

Some stakeholders are interested in whether students can follow the course, or cope 
with the structure, the workload or the pace, whether it is at the right level, or how 
much they are gaining from it. Others are more interested in whether the course was 
useful to students in the world of work. Some are concerned with how much it costs 
and whether it is worth the investment, while others again might be concerned with 
the effect of new teaching modes or new materials. 

All these approaches tell us something about the value and the worth of the course, 
but they do so from different perspectives. 

Our perspectives also have to be wide and varied. The stakeholders in our courses 
include not only past, present and future students, but their employers, their 
potential employers, and both State and Federal Government departments which 
have an interest in industry training and are currently influencing standards of 
professional development in the training field. The university is also a stakeholder, 
as its requirements and regulations, such as the Consolidated Teaching Policy, have a 
significant impact on standards and quality, and the length, size and level of the 
courses. The lecturing staff who wrote and now teach and assess the individual units 
are also very powerful stakeholders. They are the ones who have been most closely 
monitoring the units and noticing where there might be discrepancies and changing 
factors influencing the course. The lecturing staff also are dedicated to their fields of 
expertise and often have a very wide perspective across its theory and practice in 
many different contexts. A proportion of the staff have PhDs in the training field/ 
and all are engaged in research, writing and publishing in their areas. It is this very 
important fact, indeed, that seems to have been overlooked by ANT A in its attempts 
to remove universities from any significant role in the training of trainers. 

The evaluation plan 

The idea of choosing an outside evaluator began to form as a result of the staff 
response mentioned earlier. An outside evaluator would give us a different 
perspective. An evaluator selected from the market served by our courses would 
give us better insights into that market' s interest and response. The Curtin team 
began looking around at training research groups in Perth. We also began seeking 
some funding to make it possible. 

Even though our TAFE market has been shrinking and our industry trainer group 
growing and becoming identified as an important future client, we had to turn to 
TAFE to find a research organisation equipped to handle the research. The 
Department of Training and Employment in Western Australia (WA) organised two 
seminars last year, inviting interested researchers and research organisations to 
participate and present examples of their research. Most were from TAFE colleges, 
and it was an excellent chance to get an overview of the sort of research in progress 
in the State. 



We invited QRD Consulting from West Coast College at Joondalup to draw up a 
proposal. Curtin staff had worked with them before and were impressed by the 
standard of their work. 

QRD Consulting envisaged its role as a combined one, including the evaluation, the 
maintenance and improvement (defined as effectiveness and saleability) of the 
Curtin courses. They identified their objectives as follows: 

• to overview the current offerings of the program; 

• to identify key factors attracting students to, and retaining students in, the 
program; 

• to identify the level of interest or disinterest among training managers in 
TAFE colleges, government departments and industry; 

• to identify future changes for strengthening the program; and 

• to identify any untapped potential market. 

From these data they proposed to establish the value of Curtin's teacher education 
courses and their relevance to industry and government vocational education and 
training needs. They further proposed to provide short term and long term 
recommendations for improvement of the courses. The final report is planned to give 
clear insights into the strengths, weaknesses and future opportunities for the Curtin 
program. 

The researchers planned to contact industry and government trainers, academic 
institutions and staff and individual learners as the main stakeholders in the study. 

The background research stage, stage 1, will consist of familiarisation for the 
researchers with the coursework. This exists in full distance education mode in print- 
based and online formats, and is easily available. In the five courses there are in fact 
only 16 different units, so browsing the materials will not be such a large task as it 
first may seem. 

Stage 2 of the research will be a general consultation with the stakeholders, gathering 
data on which to base the major survey instruments. At stage 2 also, further 
background research will continue as Curtin lecturers are interviewed in depth to 
ascertain their perspectives on each of the units. This is expected to provide rich and 
detailed data. 

Stage 2 will also make initial contact with TAFE college management, and managers 
of those private providers where staff have undertaken one of Curtin's courses. The 
Department of Training and Employment will also be contacted at this stage, 
although officially the Department is required to follow the lead of the federal 
Department of Education, Training and Youth Affairs, and ANT A in particular. It 
has been difficult over a period of years to ascertain any opinion at all from this 
group, and we suspect it is because under present economic conditions, they are 
unprepared to pay for training. This is the situation throughout Australia at present, 
although there are indications that things might change in the future. We envisage 
that much of the contact made with management groups at this stage is more 
awareness raising than data collection. 
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The third stage will be the major survey, the nature of which will not become final 
until the completion of the interviews in stage 2. However, the survey will include 
present and past successful students, and those who have dropped out. It is hoped to 
gather data on how the course has helped, or is helping, them with their careers and 
with their sense of professionalism. It will also seek information on details of the 
strengths and weaknesses of units and where students and ex-students believe units 
should be changed. It is expected at this stage that another survey will be conducted 
with selected management stakeholders, to collect more global data on their 
perspective on the usefulness and desirability of the Curtin courses. 

An in-depth analysis and discussion of results should assist the evaluators to come 
up with a set of recommendations to be published in a detailed report. 

Conclusion 

It is an interesting experience setting up such a detailed evaluation at a time when 
our traditional market group is less interested than it has ever been. There is an 
undeserved anti-universities climate growing in the training field. On the other 
hand, university lecturers in the training field are as informed and up to date as any 
group in the country and, in Curtin's case at least, we believe we have an excellent 
product for sale. 

The evaluation may reach findings which we don't want to hear, and we may have 
to make changes we don't want to make. However, the evaluation will have two 
positive results for Curtin University. One is that, after any recommendations are put 
in place, we will have strong evidence to continue to claim that our courses are 
relevant, up to date and based on industry training needs. The second result is that 
we will have promulgated some awareness raising among a group that has been 
trying to ignore us. It is possible that they might be impressed. 
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